An orchestrated survey of available algorithms and tools for Combinatorial Testing

For functional testing based on the input domain of a functionality, parameters and their values are identified and a test suite is generated using a criterion exercising combinations of those parameters and values. Since software systems are large, resulting in large numbers of parameters and values, a technique based on combinatorics called Combinatorial Testing (CT) is used to automate the process of creating those combinations. CT is typically performed with the help of combinatorial objects called Covering Arrays. The goal of the present work is to determine available algorithms/tools for generating a combinatorial test suite. We tried to be as complete as possible by using a precise protocol for selecting papers describing those algorithms/tools. The 75 algorithms/tools we identified are then categorized on the basis of different comparison criteria, including: the test suite generation technique, the support for selection (combination) criteria, mixed covering array, the strength of coverage, and the support for constraints between parameters. Results can be of interest to researchers or software companies who are looking for a CT algorithm/tool suitable for their needs

An Extension of Category Partition Testing for Highly Constrained Systems

To ensure software is performing as intended it can be black-box or white-box tested. Category partition is a black box, specification based testing technique which begins by identifying the parameters, categories (characteristics of parameters) and choices (acceptable values for categories). These choices are then combined to form test frames on the basis of various criteria such as base choice and each choice. To ensure that the combinations of choices are feasible, constraints are introduced. While combining choices to form an each choice adequate test set it is feasible (e.g., using constrained covering arrays from combinatorial testing), the base choice criterion has not been defined to specifically account for constraints on choices. In this paper, we introduce two extensions of the base choice criterion to specifically account for complex constraints among choices. Adequate test suites of the different criteria are compared in terms of cost and effectiveness (code coverage and fault detection) on an academic and industrial case study

Extending Category Partition's Base Choice criterion to better support constraints

To ensure software is performing as intended, it can be black-box or white-box tested. Category partition is a black-box, specification-based testing technique that begins by identifying the parameters, categories (characteristics of parameters), and choices (acceptable values for categories). These choices are then combined to form test frames on the basis of various criteria such as Base Choice and Each Choice. To ensure that the combinations of choices are feasible, constraints on choices are introduced. Combining choices, while accounting for constraints, to form an each choice adequate test set is feasible (eg, using constrained covering arrays from combinatorial testing). However, the Base Choice criterion has not been defined to specifically account for constraints on choices, resulting in adverse consequences. In this paper, we introduce two extensions to the Base Choice criterion, namely, Constrained Base Choice and Extended Constrained Base Choice to specifically account for (complex) constraints on choices. We use a number of academic and industrial case studies to compare different adequacy criteria, including the new ones, in terms of cost and effectiveness at finding faults. Results show the performance of the new criteria equivalent to a 3-way combination criterion with a much smaller cost

An analysis and extension of Category partition testing for constrained systems

Software systems are overly complex and testing them is a challenging task. Testing such systems in the absence of systematic techniques results either in incomplete testing or testing with unmanageable (time, cost) number of test cases

Getting more in less: The power of single/error annotations in category partition

Category Partition (CP) [1] is a black box testing technique which is used to formalize the specification of the input domain. The process which relies on the expertise of the tester begins by identifying the parameters and environment variables on the basis of function's behaviour. The characteristics/categories of these parameters/environment variables are identified and partitioned into choices. The choices of a category are mutually exclusive and can be based on input partitioning and boundary value analysis. Thereafter, the choices are combined on the basis of various combination criteria (each choice, pairwise, all combination, base choice) to form test frames

The power of single and error annotations in category partition testing: An experimental evaluation

Category Partition (CP) is a black box testing technique that formalizes the specification of the input domain in a CP specification for the system under test. A CP specification is driven by tester's expertise and bundles parameters, categories (characteristics of parameters) and choices (acceptable values for categories) required for extensively testing the system. For completeness the choices correspond to permitted input values as well as some values to account for boundaries or robustness. These choices are then combined to form test frames on the basis of various criteria such as each choice or pairwise. To ensure that the combinations of choices are feasible and account for valid sets of user requirements, constraints are introduced to specify permitted combinations among choices, and to specify single or error choices. In a typical development environment where testing is driven by stringent deadlines a tester might have to decide how many constraints are enough to attain the maximum level of test completeness. The present work will assist a test engineer in making this decision. We conclude, on the basis of our experimental evaluation on academic and industrial case studies, that an equally effective test suite can be attained by meticulously defining error and single annotations in a CP specification while ignoring other constraints among choices. Copyright is held by the owner/author(s)